Domain Adaptation Research Articles

MFI-CD: a lightweight siamese network with multidimensional feature interaction for change detection

ABSTRACT Change detection has always played a crucial role in observing the Earth’s surface. Although visual Transformers (ViTs) have achieved excellent performance in change detection, they require high computational complexity. In addition, it does not explicitly address the differences caused by factors such as climate, lighting, and shooting angle in remote sensing (RS) image pairs. Therefore, we propose an efficient, lightweight Siamese network that considers multi-dimensional feature interaction. Firstly, we use a lightweight Transformer-based headless backbone network to extract feature information at each stage for bi-temporal images. To better capture the details and structure of images and eliminate the adverse effects of climate, lighting, and shooting angle, we design a multidimensional feature interaction method that uses spatial, channel, and amplitude dimension interaction methods after feature extraction operations at different stages. Furthermore, this approach achieves domain adaptation between bi-temporal domains to a certain extent while preserving the original semantic correspondence. Comprehensive experiments and extensive ablation studies on two common datasets, LEVIR-CD and S2Looking, have shown that our method achieves better performance with fewer parameters.

Per-class curriculum for Unsupervised Domain Adaptation in semantic segmentation

AbstractAccurate training of deep neural networks for semantic segmentation requires a large number of pixel-level annotations of real images, which are expensive to generate or not even available. In this context, Unsupervised Domain Adaptation (UDA) can transfer knowledge from unlimited synthetic annotations to unlabeled real images of a given domain. UDA methods are composed of an initial training stage with labeled synthetic data followed by a second stage for feature alignment between labeled synthetic and unlabeled real data. In this paper, we propose a novel approach for UDA focusing the initial training stage, which leads to increased performance after adaptation. We introduce a curriculum strategy where each semantic class is learned progressively. Thereby, better features are obtained for the second stage. This curriculum is based on: (1) a class-scoring function to determine the difficulty of each semantic class, (2) a strategy for incremental learning based on scoring and pacing functions that limits the required training time unlike standard curriculum-based training and (3) a training loss to operate at class level. We extensively evaluate our approach as the first stage of several state-of-the-art UDA methods for semantic segmentation. Our results demonstrate significant performance enhancements across all methods: improvements of up to 10% for entropy-based techniques and 8% for adversarial methods. These findings underscore the dependency of UDA on the accuracy of the initial training. The implementation is available at https://github.com/vpulab/PCCL.

Cross-tokamak Disruption Prediction based on domain adaptation

Abstract The high acquisition cost and the significant demand for disruptive discharges for data-driven disruption prediction models in future tokamaks pose an inherent contradiction in disruption prediction research. In this paper, we demonstrated a novel approach to predict disruption in a future tokamak using only a few discharges. The approach aims to predict disruption by finding a feature space that is universal to all tokamak. The first step is to use the existing understanding of physics to extract physics-guided features from the diagnostic signals of each tokamak, called physics-guided feature extraction (PGFE). The second step is to align a few data from the future tokamak (target domain) and a large amount of data from existing tokamak (source domain) based on a domain adaptation algorithm called CORrelation ALignment (CORAL). It is the first attempt at applying domain adaptation in the task of cross-tokamak disruption prediction. PGFE has been successfully applied in J-TEXT to predict disruption with excellent performance. PGFE can also reduce the data volume requirements due to extracting the less device-specific features, thereby establishing a solid foundation for cross-tokamak disruption prediction. We have further improved CORAL (supervised CORAL, S-CORAL) to enhance its appropriateness in feature alignment for the disruption prediction task. To simulate the existing and future tokamak case, we selected J-TEXT as the existing tokamak and EAST as the future tokamak, which has a large gap in the ranges of plasma parameters. The utilization of the S-CORAL improves the disruption prediction performance on future tokamak. Through interpretable analysis, we discovered that the learned knowledge of the disruption prediction model through this approach exhibits more similarities to the model trained on large data volumes of future tokamak. This approach provides a light, interpretable and few data-required way by aligning features to predict disruption using small data volume from the future tokamak.

Domain Adaptation for Fear of Heights Classification in a VR Environment Based on EEG and ECG

AbstractThree levels of fear of heights were detected in subjects with different severities of acrophobia, based on the electroencephalographic (EEG) and electrocardiographic (ECG) signals. The study aims to demonstrate the feasibility of a data-fusion-based method for real-time assessment of the fear of heights intensity to integrate into adaptive Virtual Reality Exposure Therapy for acrophobia. The generalization performance of classification tasks on fear states is improved by exploiting both trait-based clustering and Domain Adaptation methods. Participants were gradually exposed to increasing height levels through a Virtual Reality (VR) scenario representing a canyon. The initial severity of fear of heights, the level of distress at each height, and the anxiety level before and after the exposure were assessed through the Acrophobia Questionnaire, the Subjective Unit of Distress, and the State and Trait Anxiety Inventory, respectively. The Simulator Sickness Questionnaire was administered to exclude possible motion sickness interference in the experiment. The EEG and ECG signals were acquired through a 32-channel headset and 1 Lead ECG derivation during the exposure to the eliciting VR scenario. Four classifiers (i.e. Support Vector Machines, Deep Neural Networks, Random Forests, and k-Nearest Neighbors) were adopted in the experimental environment. Preliminary tests were performed in a within-subject experiment, achieving the best classification accuracy of $$87.1 \% \pm 7.8 \%$$ 87.1 % ± 7.8 % with a Deep Neural Network. As the cross-subject approach is concerned, three strategies, namely Domain Adaptation (DA), data fusion (combining EEG with ECG), and participant clustering (based on the acrophobia severity), were evaluated. DA resulted in the most effective strategies by determining an improvement of more than 20 % in classification accuracy. Random Forest performed the best classification accuracy for the severe acrophobia cluster with a mean of $$63.6 \%$$ 63.6 % and a standard deviation of $$ 13.4 \%$$ 13.4 % over three classes by exploiting Stratified Normalization.

Multi-Representation Joint Dynamic Domain Adaptation Network for Cross-Database Facial Expression Recognition

In order to obtain more fine-grained information from multiple sub-feature spaces for domain adaptation, this paper proposes a novel multi-representation joint dynamic domain adaptation network (MJDDAN) and applies it to achieve cross-database facial expression recognition. The MJDDAN uses a hybrid structure to extract multi-representation features and maps the original facial expression features into multiple sub-feature spaces, aligning the expression features of the source domain and target domain in multiple sub-feature spaces from different angles to extract features more comprehensively. Moreover, the MJDDAN proposes the Joint Dynamic Maximum Mean Difference (JD-MMD) model to reduce the difference in feature distribution between different subdomains by simultaneously minimizing the maximum mean difference and local maximum mean difference in each substructure. Three databases, including eNTERFACE, FABO, and RAVDESS, are used to design a large number of cross-database transfer learning facial expression recognition experiments. The accuracy of emotion recognition experiments with eNTERFACE, FABO, and RAVDESS as target domains reach 53.64%, 43.66%, and 35.87%, respectively. Compared to the best comparison method chosen in this article, the accuracy rates were improved by 1.79%, 0.85%, and 1.02%, respectively.

A Multi-Modal Egocentric Activity Recognition Approach towards Video Domain Generalization

Egocentric activity recognition is a prominent computer vision task that is based on the use of wearable cameras. Since egocentric videos are captured through the perspective of the person wearing the camera, her/his body motions severely complicate the video content, imposing several challenges. In this work we propose a novel approach for domain-generalized egocentric human activity recognition. Typical approaches use a large amount of training data, aiming to cover all possible variants of each action. Moreover, several recent approaches have attempted to handle discrepancies between domains with a variety of costly and mostly unsupervised domain adaptation methods. In our approach we show that through simple manipulation of available source domain data and with minor involvement from the target domain, we are able to produce robust models, able to adequately predict human activity in egocentric video sequences. To this end, we introduce a novel three-stream deep neural network architecture combining elements of vision transformers and residual neural networks which are trained using multi-modal data. We evaluate the proposed approach using a challenging, egocentric video dataset and demonstrate its superiority over recent, state-of-the-art research works.

Cataract-1K Dataset for Deep-Learning-Assisted Analysis of Cataract Surgery Videos

In recent years, the landscape of computer-assisted interventions and post-operative surgical video analysis has been dramatically reshaped by deep-learning techniques, resulting in significant advancements in surgeons’ skills, operation room management, and overall surgical outcomes. However, the progression of deep-learning-powered surgical technologies is profoundly reliant on large-scale datasets and annotations. In particular, surgical scene understanding and phase recognition stand as pivotal pillars within the realm of computer-assisted surgery and post-operative assessment of cataract surgery videos. In this context, we present the largest cataract surgery video dataset that addresses diverse requisites for constructing computerized surgical workflow analysis and detecting post-operative irregularities in cataract surgery. We validate the quality of annotations by benchmarking the performance of several state-of-the-art neural network architectures for phase recognition and surgical scene segmentation. Besides, we initiate the research on domain adaptation for instrument segmentation in cataract surgery by evaluating cross-domain instrument segmentation performance in cataract surgery videos. The dataset and annotations are publicly available in Synapse.

A Comprehensive Exploration of Fidelity Quantification in Computer-Generated Images

Generating realistic road scenes is crucial for advanced driving systems, particularly for training deep learning methods and validation. Numerous efforts aim to create larger and more realistic synthetic datasets using graphics engines or synthetic-to-real domain adaptation algorithms. In the realm of computer-generated images (CGIs), assessing fidelity is challenging and involves both objective and subjective aspects. Our study adopts a comprehensive conceptual framework to quantify the fidelity of RGB images, unlike existing methods that are predominantly application-specific. This is probably due to the data complexity and huge range of possible situations and conditions encountered. In this paper, a set of distinct metrics assessing the level of fidelity of virtual RGB images is proposed. For quantifying image fidelity, we analyze both local and global perspectives of texture and the high-frequency information in images. Our focus is on the statistical characteristics of realistic and synthetic road datasets, using over 28,000 images from at least 10 datasets. Through a thorough examination, we aim to reveal insights into texture patterns and high-frequency components contributing to the objective perception of data realism in road scenes. This study, exploring image fidelity in both virtual and real conditions, takes the perspective of an embedded camera rather than the human eye. The results of this work, including a pioneering set of objective scores applied to real, virtual, and improved virtual data, offer crucial insights and are an asset for the scientific community in quantifying fidelity levels.

Deep Siamese domain adaptation convolutional Neural Network optimized with Honey Badger Algorithm for Fetal Arrhythmia Detection and Classification from ECG Signals

Deep Siamese domain adaptation convolutional Neural Network optimized with Honey Badger Algorithm for Fetal Arrhythmia Detection and Classification from ECG Signals

Fault severity recognition in axial piston pumps using attention-based adversarial discriminative domain adaptation neural network

Axial piston pumps are the ‘hearts’ of hydraulic systems whose fault recognition is necessary for the safety and reliability of hydraulic equipment. These pumps operate under different operating conditions and the fault recognition model trained at one operating point cannot be applicable at another operating point due to the problem of domain shifts. This paper proposes a transfer learning method for the fault severity recognition of axial piston pumps based on adversarial discriminative domain adaptation fused with a convolutional channel attention module. First, a convolutional neural network is pre-trained with labeled vibration data from the source domain, and a convolutional channel attention module is added to assign weights to different convolution kernels. Second, the trained source model is transferred to the target domain, and its parameters are updated by an adversarial training process between the labeled source data and the unlabeled target data. Finally, vibration data are collected from an axial piston pump at different fault levels under various operating conditions to validate the proposed method. Experimental results indicate that the proposed method achieves an average recognition accuracy of 98.3% and outperforms some other transfer learning methods by a large margin.

Distance-guided domain adaptation for bearing fault diagnosis under variable operating conditions

Abstract Transfer learning (TL) has demonstrated effective application in diagnosing bearing faults under varying operating conditions. The current TL methods achieve domain alignment by minimizing the variation in the marginal distribution of data between the source and target domains in the feature space. However, this approach yields incomplete similarity, leading to domain shift and a decrease in diagnostic performance. To overcome this problem, this paper proposes a new distance-guided domain adaptation (DGDA) method that consists of two modules: Deep domain adaptive correlation alignment (Deep CORAL) combined with Joint Maximum Mean Discrepancy (JMMD) for guided domain adaptation. Deep CORAL employs nonlinear transformations to synchronize second-order statistical correlations across source and target domains, thus ensuring feature-level alignment between these domains. JMMD is utilized to align the joint distribution of input features and output labels within the activation layer in the deep network, thereby bolstering domain alignment. Building on this, we propose a network structure that merges ResNet and BiLSTM, powered by wavelet kernels, serving as a feature extractor. This structure is designed to learn domain-invariant features and incorporates attention mechanisms to amplify important information while diminishing the impact of redundant data. An analysis of bearing experiments is used to demonstrate the effectiveness of this method, and the proposed method significantly outperforms several popular methods in diagnostic performance.

A few-shot based phase-batch multi-layer domain adaptation pattern recognition method

Deep transfer learning has been widely applied in the field of intelligent fault diagnosis. However, existing deep transfer learning-based diagnostic methods struggle to train reliable diagnostic models when there is a lack of data and significant distribution differences between the two domains. To address this issue, a few-shot based phase-batch multi-layer domain adaptation pattern recognition method is proposed. This method simultaneously measures the feature distribution differences of both the fully connected layers and the classification layers, thus better correcting the data domain bias. Additionally, a phase-batch training strategy and pseudo-label learning are employed to improve the convergence speed and stability of the training process. The proposed method is validated on two public datasets, Jiang Nan and Paderborn University, as well as a dataset obtained through independent experiments. It is compared with traditional feature-based transfer learning methods, the results show that the proposed method achieves higher diagnostic accuracy, faster convergence, and greater stability. Furthermore, its superior diagnostic performance in the few-shot scenario is demonstrated through experiments on a self-collected dataset.

Transferable mapping shift network for unsupervised domain adaptation using in vibration signal fault diagnosis under variable conditions

Transferable mapping shift network for unsupervised domain adaptation using in vibration signal fault diagnosis under variable conditions

Virtual reality games for cognitive rehabilitation of older adults: a review of adaptive games, domains and techniques

In recent decades, the senior adults population worldwide has increased, as well as the medical conditions related to aging, such as cognitive decline. Virtual reality (VR) games are a valuable addition to conventional cognitive rehabilitation as they increase engagement to the therapy through customization, socialization, immersion, and feedback. This review, performed according to PRISMA protocol, addresses the following questions: How VR games have been used for cognitive rehabilitation?, What cognitive domains have been addressed by VR games and in which populations have these games been used?, Which features have been considered to improve engagement in VR games for cognitive rehabilitation?, How is the difficulty adjustment of exercises carried out in VR games for cognitive rehabilitation?. We found 25 scientific works related to these questions, 92% of them treating one cognitive domain at a time, despite the fact that the related literature recognizes the value of training multiple domains simultaneously. Our review indicates that, despite the existence of serious VR games for working memory training, such as those described in Flak et al. (Front Psychol 10:807, 2019. https://doi.org/10.3389/fpsyg.2019.00807), to our knowledge, there are no applications that simultaneously address multiple cognitive domains and incorporate dynamic difficulty adjustment, which are important to ensure ecological validity of therapy and therapy adherence, respectively. In addition, we found that games themselves could be used to monitor the user’s progression. It is also important to determine the impact of multiplayer interactions in the game, test difficulty adjustment approaches that use physiological variables, and define difficulty-skill relationships aligned with the user’s preferences. This paper concludes that the main barriers to implement dynamic difficulty adjustment in VR games for cognitive rehabilitation are: (i) the absence of metrics to estimate when the game offers to the players a challenge adapted their skills, and (ii) the lack of a conceptual framework that integrates relevant theories such as state of flow, cognitive load, cognitive rehabilitation, and feedback systems.

Reusability report: Uncovering associations in biomedical bipartite networks via a bilinear attention network with domain adaptation

Reusability report: Uncovering associations in biomedical bipartite networks via a bilinear attention network with domain adaptation

SFDA-CD: A Source-Free Unsupervised Domain Adaptation for VHR Image Change Detection

Deep models may have disappointing performance in real applications due to the domain shifts in data distributions between the source and target domain. Although a few unsupervised domain adaptation methods have been proposed to make the pre-train models effective on target domain datasets, constraints like data privacy, security, and transmission limits restrict access to VHR remote sensing images, making existing unsupervised domain adaptation methods almost ineffective in specific change detection areas. Therefore, we propose a source-free unsupervised domain adaptation change detection structure to complete specific change detection tasks, using only the pre-trained source model and unlabelled target data. The GAN-based source generation component is designed to generate synthetic source data, which, to some extent, reflects the distribution of the source domain. Moreover, these data can be utilised in model knowledge transfer. The model adaptation component facilitates knowledge transfer between models by minimising the differences between deep features, using AAM (Attention Adaptation Module) to extract the difference between high-level features, meanwhile we proposed ISM (Intra-domain Self-supervised Module) to train target model in a self-supervised strategy in order to improve the knowledge adaptation. Our SFDA-CD framework demonstrates superior accuracy over existing unsupervised domain adaptation change detection methods, which has 0.6% cIoU and 1.5% F1 score up in cross-regional tasks and 1.4% cIoU and 1.9% F1 score up in cross-scenario tasks, proving that it can effectively reduce the domain shift between the source and target domains even without access to source data. Additionally, it can facilitate knowledge transfer from the source model to the target model.

A universal fault diagnosis framework for marine machinery based on domain adaptation

In the digital and automated era, the significance of machine health monitoring has been amplified, especially for marine machinery, which plays a pivotal role in the sustainable advancement of modern industries. Two salient challenges currently hampering fault diagnostics are spotlighted: firstly, the variability in mechanical operating conditions gives rise to inconsistent data distributions, thereby prompting domain shift. Secondly, unobserved in training datasets, testing scenarios may unveil unpredictable, unknown fault modes, culminating in class discrepancies. To effectively tackle these intertwined challenges, we introduce the Two-Stage Universal Domain Adaptation (TSUDA) — a novel fault diagnosis methodology. TSUDA adeptly differentiates new fault types from labeled known counterparts by leveraging a composite transferability metric. Furthermore, it deploys a parameter-adaptive unsupervised algorithm to discern the count of emerging fault types. Rigorous experiments on datasets from marine diesel engines, emblematic of marine machinery, demonstrated TSUDA's superiority over conventional domain adaptation techniques in diverse diagnostic challenges, registering accuracy rates surpassing 90%. While the effectiveness of TSUDA is authenticated within marine machinery contexts in this investigation, its architecture suggests potential applicability in a broader spectrum of mechanical fault diagnostic scenarios.

Cross-domain knowledge collaboration for blending-target domain adaptation

Unsupervised domain adaptation (UDA) avoids expensive data annotation for the unlabeled target domains by fully utilizing the knowledge of existing source domain. In practice, the target data are usually highly heterogeneous that mix multiple latent domains. And sometimes the source data involve the user private information which is forbidden to access directly. To this end, this paper tackles the blending-target data under the source-available setting, and for the first time under the source-free setting. Specifically, we devise a novel Cross-domain Knowledge Collaboration (CdKC) framework, which mainly includes a prediction network and an adaptation network. The complementarity of two networks are exploited to explore the intrinsic structure in targets. Specifically, CdKC is capable of learning domain-invariant space, and disentangling domain-specific feature simultaneously to boost the UDA performance greatly. A total of 12 tasks are conducted on three visual datasets to verify the superior performance of CdKC by comparing with the state-of-the-art models designed under 4 different UDA settings. The experiments demonstrate that the accuracy of CdKC model still exceeds D-CGCT 0.4% on Office dataset and 1.2% on Office–Home dataset although D-CGCT can access source data and domain labels of targets that CdKC cannot do. It verifies the effectiveness of CdKC although under much looser source and target domain restrictions.

An adversarial transfer learning method based on domain distribution prediction for aero-engine fault diagnosis

The utilization of transfer learning enables effective realization of the transferability of aero-engine fault diagnosis models across disparate states. However, the gradual degradation of engine performance and the complex and variable flight states lead to continuous changes in data distribution. The common transfer learning methods employ discrete divisions of data domain distributions, which are insufficient to cope with the continuous changes of the domain. To accomplish transfer learning towards a continuous and multi-dimensional target domain, we propose a continuous domain distribution adversarial network (CDDAN) based on deep domain adversarial network. The method defines the number of effective cycles directly associated with engine degradation as a continuous domain index. When the domain index is extended to multiple dimensions, a hybrid gaussian model is introduced to represent distribution prediction within multi-dimensional continuous domains. Subsequently, we validate the feasibility and superiority of our method using datasets generated based on twin-spool turbofan engines. In comparison to other transfer learning methods, the proposed method achieves superior effect in continuous domain adaptation. The experimental results demonstrate that this method exhibits adaptability to failures in the diagnosis method caused by performance degradation and changes in flight state of the aero engine, while remaining independent of target domain data annotation. This advantage becomes particularly apparent in scenarios with limited target domain data and multi-state fault diagnosis.

CAT-DTI: cross-attention and Transformer network with domain adaptation for drug-target interaction prediction

Accurate and efficient prediction of drug-target interaction (DTI) is critical to advance drug development and reduce the cost of drug discovery. Recently, the employment of deep learning methods has enhanced DTI prediction precision and efficacy, but it still encounters several challenges. The first challenge lies in the efficient learning of drug and protein feature representations alongside their interaction features to enhance DTI prediction. Another important challenge is to improve the generalization capability of the DTI model within real-world scenarios. To address these challenges, we propose CAT-DTI, a model based on cross-attention and Transformer, possessing domain adaptation capability. CAT-DTI effectively captures the drug-target interactions while adapting to out-of-distribution data. Specifically, we use a convolution neural network combined with a Transformer to encode the distance relationship between amino acids within protein sequences and employ a cross-attention module to capture the drug-target interaction features. Generalization to new DTI prediction scenarios is achieved by leveraging a conditional domain adversarial network, aligning DTI representations under diverse distributions. Experimental results within in-domain and cross-domain scenarios demonstrate that CAT-DTI model overall improves DTI prediction performance compared with previous methods.